perf: switch saturation tests to static/1KB logs and reduce loadgen to SUT ratio to 2:1#2643
Draft
cijothomas wants to merge 4 commits into
Draft
perf: switch saturation tests to static/1KB logs and reduce loadgen to SUT ratio to 2:1#2643cijothomas wants to merge 4 commits into
cijothomas wants to merge 4 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2643 +/- ##
==========================================
- Coverage 86.03% 86.03% -0.01%
==========================================
Files 720 720
Lines 273264 273270 +6
==========================================
Hits 235095 235095
- Misses 37645 37651 +6
Partials 524 524
🚀 New features to boost your workflow:
|
…atio to 2:1 - Switch saturation tests to use static data source with 1KB log bodies (pre_generated strategy) for more realistic payload sizes - Reduce loadgen:engine core ratio from 3:1 to 2:1 across all saturation configs - Make data_source and log_body_size_bytes templatable in loadgen config - Existing non-saturation tests are unaffected (defaults unchanged) Validated locally on 16-core machine (4 runs each, 1/2/4 engine cores): - 2:1 ratio saturates engine to 96-100% CPU at 1-2 cores - 33% fewer loadgen cores needed - 3.75x higher data throughput (514 MB/s vs 137 MB/s at 2 cores)
cijothomas
force-pushed
the
cijothomas/saturation-static
branch
from
399642f to
6354a14
Compare
…late The saturation template sets data_source=static and log_body_size_bytes=1024, but these variables were not forwarded through the test steps template to the loadgen config template. Add the passthrough so the loadgen actually uses static/1KB logs as intended.
|
This pull request has been marked as stale due to lack of recent activity. It will be closed in 30 days if no further activity occurs. If this PR is still relevant, please comment or push new commits to keep it active. |
github-actions
Bot
added
stale
|
This pull request has been marked as stale due to lack of recent activity. It will be closed in 30 days if no further activity occurs. If this PR is still relevant, please comment or push new commits to keep it active. |
Resolve conflict in df-loadgen-steps-docker.yaml: keep both data_source/log_body_size_bytes from this branch and exporter_extra_config from main.
The build_body_pool function previously padded log bodies by repeating the SAME template text to reach the target size. For 1KB bodies, a ~150-char template was repeated ~7x, creating unrealistically compressible content (7.4:1 per body vs ~1.5:1 for real logs). Changes: - Cycle through DIFFERENT templates to fill each body instead of repeating one - Add deterministic 'seq=NNNN' prefix so no two pool entries are identical - Expand pool from 50 to 512 entries (matches default batch_size) to avoid duplicate bodies within a batch inflating cross-record compression Per-body compression improves from 7.4:1 to 3.1:1 (much closer to real structured log text). Batch-level compression stays at ~18.7:1 which is realistic for OTLP batches with shared schema/keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Summary
pre_generatedstrategy) for more realistic payload sizes compared to tiny semantic-convention bodies.seq=Nprefix. This produces realistic per-body compression ratios (~3:1) instead of the previous approach that repeated a single template ~7× per body (~7:1).data_sourceandlog_body_size_bytestemplatable in loadgen config with backward-compatible defaults.How are these changes tested?
Validated locally on 16-core machine:
Both tests confirm the engine is saturated with the 2:1 ratio. Full orchestrator reports generated successfully (shutdown + SQL report bugs that existed on the old branch were resolved by merging latest main).
Are there any user-facing changes?
No. Internal perf tests and traffic generator only.